<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> 
    <title>Instructions for speech quality evaluation</title>
    <style type="text/css">
body, input, select, td, li, div, textarea, p {
	font-size: 11px;
	line-height: 16px;
	font-family: verdana, arial, sans-serif;
}

body {
	margin:5px;
	background-color:white;
}

h1 {
	font-size:16px; 
	font-weight:bold; 
}

h2 {
	font-size:14px; 
	font-weight:bold;
}
    </style>
  </head>

  <body>
    <h1 align="center">Instructions for speech quality evaluation</h1>
    <h2>Introduction</h2>
    <p>Your task is to evaluate the subjective quality of the speech from short (2-3 second) audio files. Each HIT can be completed in 90 seconds.</p>

    <h2>Payment</h2>
    <p>We have methods that analyze the consistency of your answers with respect to themselves, to those of your fellow workers and to references we know to be accurate. We will use these methods to rank the submitted assignments according to quality.
    <p>For this experiment we will pay a base reward of $0.10/HIT for every accepted HIT. We have made available a set of 12 different HITs. You will receive a bonus of:</p>
    <ul>
      <li>$0.10/HIT (for a total of $0.20/HIT) if you submit all 12 HITs or</li>
      <li>$0.20/HIT (for a total of $0.30/HIT) if you submit all 12 HITs and your results are among the top 50% or</li>
      <li>$0.30/HIT (for a total of $0.40/HIT) if you submit all 12 HITs and your results are among the top 10%.</li>
    </ul>

    <p>Bonuses will be paid up to 7 days after submission, because we can only rank the submissions once we have a statistically significant number of answers. The base reward will always be paid within 24 hours of submission.</p>

    <h2>Instructions</h2>
    <p>Each file should be given a score according to the following scale, known as the MOS (mean opinion score) scale:</p>

      <table>
        <tbody>
          <tr>
            <td align="center" width="100"><b>Score</b></td>
            <td width="200"><b>Quality of the Speech</b></td>
            <td width="300"><b>Level of Distortion</b></td>
          </tr>
          <tr>
            <td align="center">5</td>
            <td>Excellent</td>
            <td>Imperceptible</td>
          </tr>
          <tr>
            <td align="center">4</td>
            <td>Good</td>
            <td>Just perceptible, but not annoying</td>
          </tr>
          <tr>
            <td align="center">3</td>
            <td>Fair</td>
            <td>Perceptible and slightly annoying</td>
          </tr>
          <tr>
            <td align="center">2</td>
            <td>Poor</td>
            <td>Annoying, but not objectionable</td>
          </tr>
          <tr>
            <td align="center">1</td>
            <td>Bad</td>
	    <td>Very annoying and objectionable</td>
          </tr>
        </tbody>
      </table>

      <p>The following references illustrate the meaning of each score. Please note that you will encounter many other types of noise and distortion. Therefore, these examples do not exhaust the range of conditions you can expect to hear.</p>

      <h2>Reference Samples</h2>
      <p>The following recording represents clean speech with imperceptible noise or distortion, which is given a reference score of <b>5.0</b>.</p>
        
<object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000" codebase="http://fpdownload.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=7,0,0,0" height="24" align="middle" width="400">
<param name="quality" value="high">
<param name="FlashVars" value="soundFile=data_qual/mos5.0.mp3">
<param name="movie" value="player.swf?titles=">
<embed src="player.swf" pluginspace="http://www.macromedia.com/shockwave/download/index.cgi?P1_Prod_Version=ShockwaveFlash"
 type="application/x-shockwave-flash" flashvars="soundFile=data_qual/mos5.0.mp3" quality="high" height="24" width="400">
</object>

      <p>The following represents the best possible quality which can be obtained with a conventional telephone, and has a reference score of <b>4.5</b>.</p>
        
<object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000" codebase="http://fpdownload.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=7,0,0,0" height="24" align="middle" width="400">
<param name="quality" value="high">
<param name="FlashVars" value="soundFile=data_qual/mos4.5.mp3">
<param name="movie" value="player.swf?titles=">
<embed src="player.swf" pluginspace="http://www.macromedia.com/shockwave/download/index.cgi?P1_Prod_Version=ShockwaveFlash"
 type="application/x-shockwave-flash" flashvars="soundFile=data_qual/mos4.5.mp3" quality="high" height="24" width="400">
</object>

      <p>This file contains speech corrupted by background noise, and has a reference score of <b>2.5</b>.</p>
        
<object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000" codebase="http://fpdownload.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=7,0,0,0" height="24" align="middle" width="400">
<param name="quality" value="high">
<param name="FlashVars" value="soundFile=data_qual/mos2.5.mp3">
<param name="movie" value="player.swf?titles=">
<embed src="player.swf" pluginspace="http://www.macromedia.com/shockwave/download/index.cgi?P1_Prod_Version=ShockwaveFlash"
 type="application/x-shockwave-flash" flashvars="soundFile=data_qual/mos2.5.mp3" quality="high" height="24" width="400">
</object>

      <p>Finally, this is an example of significantly distorted speech, with a reference score of <b>1.5</b>.</p>

<object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000" codebase="http://fpdownload.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=7,0,0,0" height="24" align="middle" width="400">
<param name="quality" value="high">
<param name="FlashVars" value="soundFile=data_qual/mos1.5.mp3">
<param name="movie" value="player.swf?titles=">
<embed src="player.swf" pluginspace="http://www.macromedia.com/shockwave/download/index.cgi?P1_Prod_Version=ShockwaveFlash"
 type="application/x-shockwave-flash" flashvars="soundFile=data_qual/mos1.5.mp3" quality="high" height="24" width="400">
</object>

      <h2>Approval/Rejection Policy</h2>
      <p>To obtain accurate results, we strongly recommend that you wear headphones and work in a quiet environment, otherwise you might not be able to discriminate between files with clearly different features.</p>
      <p>Your results will be collected and evaluated for consistency. We (the requesters) have an estimate of each file's subjective quality that conforms with the references above. Thus, we can detect if someone submits random scores or does not rate according to these instructions, which can lead to work being rejected. You can rest assured that your work will be approved if you rate according to the chart and examples above.</p>
      <p>Answers will be either reviewed or automatically approved within 24 hours.</p>

</body>
</html>
